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ABSTRACT • - > 

„ Real test data of unknown structure were analyzed 

using both a unidimensional and a multidimensional latent trait model 
in an attempt to determine the underlying components of the test. The 
models used were the threeyparameter logistic model and a 
multidimensional -extension of the two-parameter logistic model. The 
basic' design for the analysis- of the data was to start with the 
unidimensional model, evaluate the fit of the model to the data, then 
increase-* the dimensionality of the model and perform the same 
analyses. The dimensionality of the model was to be increased until 
deviations from fit were acceptably small. Once acceptable fit was 
obtained, the item parameter estimates were to be analyzed to 
determine the ^structure of the ability components required by the 
test. The results of the analyses indicated that a two-dimensional 4 
solution yielded no better fit than a unidimensional solution, 
although the test data were selected to be multidimensional. From the 
results of the analyses it was concluded that the data had a- 
difficulty factor that was sufficiently great as to dominate the 
o-ther, intended factors. (Author) 
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" There are ^many situations in the practice of testing in which data are 
collected from 'tests that measure more than # a single trait. Indeed j^-tf^w^T 1 
"strong argument can probably be made that dimensionally complex measuring 
devices are the rule rather than the exception^ If the items in the test ' 
are dichotomously scored, the analysis of these data poses special problems, 
since the Methodology for the analysis of these data/ ia not well developed. 
The purpose of this paper is to demonstrate the Capabilities pf a relatively 
new methodology that has been developed in the last 25 yeass specifically 
to analyze test data. This methodology is variously called item response 
theory (IRT), item characteristic curve theory (ICC), or latent- trait 
theory. In addition to the us£ of the already well known one-dimensional 
IRT models, this. paper will introduce the use of;a multidimensional extension 
of the two-parameter logistic* model (M2PL) that is a sjyprfal case x)f an 
extremely general model proposed by Rasch (1961). Using these analysis 
models, an attempt will be made to describe a set of tes€ data of unknown 
structure and determine the underlying components of the test^ 



Design or the Analysis 



The data that were analyzed in this study were selected from the responses 
of individuals, to items from tests available at the American College Testing 
Program (ACT). Bob'Brennan selected the response data used in the. study 
specifically for this symposium. He was instructed to prouuce a 
multidimensional data-set from the responses to existing tests that would 
serve as a good check on the capabilities of severaL.multidimensional 
analysis techniques. The result was a set of 2794 dichotomous response 
strings of 50 items in length. No information about the characteristics of 
the data was given to the participants of the symposiunf before this 
presentation other than that given above. * * 

The basic design for the analysis of the test data was to start, with a - 
one-dimensional model, evaluate the fit of the model to the data, then 
increase the dimensionality of the model andflaerform the same analyses 
until deviations from fit became negligible. Wnce acceptable fit was 
obtained, the item parameter estimates that were^jcpmputed using the model 
would then be* analyzed to determine the structure of the ability componej/ts 
required by the t,est. N „ 
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. . Two different one-dimensional models ware used in the study. The 
thfequatiot 6 thr6e lo 8 stic ™ odel (3PL) (Birnbaum, 1968) given by 

' 1 + e 1 3 1 

where PfX^ll a.,b\ .c.^..) is t& probability of a correct response to Item 
i by person j, xV. is the item score (0 or ^obtained by Person j on Item 
i, a ± is the item' discrimination paramster, b £ is the item difficulty 
parameter, c. is the lower asymptote, 9 is the ability of Person j,.D is " 

V ^?J°?T Sta f hL?* qUl !i* d t0 ™ ake thG fuhction similar /to the normal ogive 
model (Lord, 1952),, and e is the constant 2.718...'. . • 

The second one-dimensional model that was. used in the study- was the 
^wo-paramete^ logistic model (2PL) (Birnbaum, 1968) given by 
» * 

' . . d i + a i 9 j ' r 

1 + e 1 1 J* 

where d. is the logistic .intercept term' equal to -b.a. and all of the^ther 
symbols are defined as above. The model ^Equation 2 is given in- the 
slope.- intercept farm rather than with the usual exponent, a.(6.-b.), so as " 
to be more readily compared with the multidimensional model* J 1 

The multidimensional model use^ in this study is a special case of the 
general Rasch model (Rasch, 1961). It is also a multivariate extension of 

equation Parameter l0giStiC model (M2PL) ' T he M2PL model is 8 ive n by the 

. 1 + e 1 - 1 . (3) \ 

where a £ is a vector of discrimination-parameters "for Item i, 6. is a 
IbovT^ abiUty Parameters for Pe "° n J' and - the othe * symbols are defined 

. The fit of the models to the data used in this study was determined 
using the residual covariance matrix. This matrix was obtained by computing 
UiAcovanance between*the differences between the item response and the 
predicted probability of correct response based on the item and person 
parameters estimated from the data. The distribution of the residual 
covaxiances was then, compared to the distribution expected if no relationship 



were present between the items. If the distributions were sufficiently 
different, it was hypothesized that added dimensions were needed' in the 
model to explain the responses. • < § ^ , * 

In addition to the direct analysis of the real data, supplementary 
'analyses were performed on simulated testvdata that were. generated to have 
the same three-parameter logistic item parameters as the real data. v These 
analyses were performed *to serve afe a unidimensional basis for comparison 
in interpreting the analyses performed qn the rea^ data. 

Descriptive statistics were T also computed on the item and person 
parameters obtained frdm the IRT analyses. These statistics included the 
jneansj standard^deviations / and correlations between the parameter estimates. 
Wherfc appropriate, distributions of the item parameters were constructed. 



Results 



One-Dimensional Analyses 



Two different IRT models were used to analyze the test data assuming a* 

one-dimensional latent space. These rapdels were the three-parameter logistic 

(3PL) model and th6 two-parameter 'Logistic (2PL) model. "The results<of the 

3PL analysis to both the real and simulated data will be presented first. ✓ 

# * 

T hree-Parameter Logistic Analysis The 3PL analysis of the real^ data ^ 
was performed usin^ the 1982 version of LOGIST (Wingersky, JJarton, and > 
'Lord, 19&2) . The atem parameter estimates obtained from the analysis and 
traditional item statistics are presented in Table 1. Note' that -the items 
were originally arranged on the test according to the proportion of correct. h 
responses. This order is .also maintained by thq 3PL b-parameter estimates 
except in the, cases where the a-parameter estimates are small, or the 
c -parameter estimates are large. The magnitude of the. item parameter estimates 
are typical of thos^ found on a standardized test,' with the exception of 
Items. 16, 18, 30, atxd 32, which have unusually high c-parameter estimates. 
An analysis of the distribution of the v a- and b-parameter estimates indicated 
that the a-paranieters tended to have a few more* low values than. expected, 
and that the b-parameters had an essentially rect.angular distribution. * 

• \ ' 

- % The correlations between the parameter estimates and- traditional item 
^statistics are given' in Table 2. This table contains several noteworthy 
correlations'. First, the correlation between the ^-parameter jsstijmates and \ 
the biserial correlation discrimination inctex is only .02. The small 
magnitude of this correlation is quite unusual, since^for unidimensional 
test data these values should be quite highly related^ The unusual nature 
of this correlation is .furthet emphasized by the fact that both the a-values 
and the r n „-values are correlated witah the proportion correct difficulty 

' Bib i * , 

index (P), but the correlations are opposite in sign. The correlations y 
indicate that the a-parameter and rg^g are related to independent components 
of the variation in P. 
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Table 1 

. LOGIST 3PL Item Parameter Estimate 
'for the AERA Symposium Data- 



Parameter 



N 



■1 
2 
3 
4 
5 
6 
7 
8 
9 

10 
11 
12 
13 
14 
15 
16 
17 
18 
19 
20 
21 
22 
23 
24 
25* 



> 



" .65 
.'46 
.53- 
.98 
.88 
.33 
.96 
'1.10 
1.07 
.52 
.83, 
.93 
.72- 
.60 
. ,73 
1.29 
1.21 
.1.11 
.68 
.40. 
.91 
1.^08 

:67 
.56 
1.32 



b 


c 


• ■ P 


-2.64 


a 

■ . 16 


.93 


-2.3.1 


.16 


.86 


-2.10 v 
-1.41 


a.16 t 


.86 


.16 


.86 


.-1.30 


.16 


. .83 


.-2.49 


.16 " 5 


.82 


-1.03 ' v . 


. -.16 


.80 


'-..96 r * 


.16 


.79 


- .96 1 


.16 


* '.79 


-1.45- 


:i6 


".79 


- .99 


.16 s " 


-77 


- .86 


.16 • 


.'76 


- .92- 


^16 


.76> 


-1.08 


.16 ' 


.75 


.90 


.i6 , 


.75 


.11 * 


.50 


.74 


- .79 


.03 ' 


.73 


.A3. • r 


' .50 • 


.72 


r .79 / . 


.16 


- .72 


-1.11 


.16 


.71 


- .57 


.16 • 


.70 


- .50 » 


• 16 


.70 


- .66 


V 16 


.70 


- >62 


*16 


.68 


- .22 


.23 


• .67 



L BIS 



.51 
.39 
.46 
.66 
.61 
.34 
• .64 
.69 
.68 
.,44 ■ 
."60 
.62 
.58 •■ 
.50 
.56 
.49 
.75 . 
.48 
.54 
.38 
.•62 ' 
.67 
.52 
•47. 
.67 
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. Parameter 






Item 


3 

„ 9 


b 




P 


mBIS 


OA 

zo 


i nn 
1 . uu. 


- .23 


.19" 


Zc 
.00, 


- 

.OZ 


97" 
Z/ 


ZO 

♦ oy * 


- .36 




Z/ 
.OH 


.52 


zo # 


. VZ 


-* 1 .29 


.16 


ZO 
.00 


.55 


00 

zy 


3 HO 

— i.Uz .* 


- .25 


.13 


z o 
.63 


£ z 

.66 




1 ♦ 1U 


.93 


.43 ^ 


.57 


■ .»37 


01 


1 10 

1 ♦ lo 


.09 


.17 


i 

c Z * 
.00 


z o 
.63 


0 "J 


A A 


1.47 


.44 


c Z 

.56 


o o 
*2o 


00 


QO 4 
. OZ 


r 

• 15 .. 


.15 


♦ 5h « 


c Z 

.56 


OA 
OH 




•41 


.22 


.53 


.52 


OQ 


0 1 

. y i 


.26 




.52 




JO 


i no 
* < 


.4\ 


. 13 • 


/. z 
.4o 


zo 

. OZ 


07 


1 0 

. /o 


. • 1.38 


.32 


/ z 


♦ 34 


OQ 

JO / 


1 no 
1 , Uo 


.79 • 


.'22 


/ A 
♦ HH 


.50 


OQ 

oy 


ZO 
• v OZ 


.42' • 


.05 


' / / * 
. HH ' 


♦ 51 


An 

hu 


W 


' -.59 


. 12 


.43 


.54 


A 1 

Hi 


c o 
.53 


1 .27 


,14 


-.37 


. .3*8 


HZ 


1 OA 

1 . oU 


1.22 


.23 


.37 


/ z 

.HO 


AO 
HO 


1. A/. 

l"i UH 


1.15 


.18 


OA 

.3h 


AZ 
. HO 


AA 
HH 


z . uu 


1 27 


' .23 


Ofc 


07 
. O / 


45 


kOO 


, 1.33 


• .14 


.29 


, .43 


46 


■1.17 


1.85 " 


.21 


.27 ' 


.28 


47 


.83 


. » 1.45 • 


; .09 . ' 


.26 


.45 


48 ' 


1.19 ' 


1.64 • 


' ; .15 


.24 


'.36 




1 . O 1 


1.89 


• 

.17 


00 
. Z 0 


OQ 

♦ zy 


50. 


1.50 


2.57 


.13 


.15 * ' 


' ' .13 


x 


.91 


. ( - ,10 


.18 . 


.60 


.50 


SD 


.31 


"'1.22 


.10 


.20 


• .13 
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Tabled 



Correlations Between 3PL Item Parameter Estimates 
and Traditional Item Statistics 



7 ; 

Item Statistics 



BIS 



a 
b 
c 
P 



.54* 



v .21 
.24 



-.46* 
-.94* 
.03 



.02 
7.50* 
-.30* 
-.54* 



BIS 



Note: The correlations with asterisks are significant beyond the .05 level. 

Other correlations of interest are the -.3C correlation between the 
c-parameter value and the r gIS values, and the correlation of .54 between 

the a- and b-parameters. Neither of these correlations are surprising. 
Items with high guessing levels would be expected to have low values for 
r BIS' and the a ~ and k*P arametei:s have beep found to be correlated in other 
studies. " 

In order to determine whether the 3PL model fit the data reasonably 
well,, the residual cov2riance matrix* was computed'. .This matrix contains 
the covariance between the residuals for each item and every other item. In 
order to interpret this mattix, a frequency distribution was constructed 
, using the values from above t lie, main diagorial of the matrix. This distribution , 
is showii in Figure 1. Descriptive statistics for the distribution are also 
shown in Figure 1. 

j 

As can easily be seen from the figure, the distribution of residual 
covariances is quite positively skewed, and the majority of the values fall 
between 0 and -.1. This range contains appsoximately 77°/- of the values. 

If. the data truly fit # this unidimensional model, we would have expected 
a symmetric distribution around zero with a standard deviation of approximately 
.019. The standard deviation of the observed distribution is .037, substantially 
larger than this value. 
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Since little background information 'is available on how to interpret 
the residual-vcovariance matrix; a simulated data set was produced for the 
purpose of obtaining comparative analysis results based on unidimensional 
$ata. This data set was generated using the 3Pl/ model and the itam parameter 
estimates obtained from the analysis of the ACT data. A total of 1000 * 
simulated examinees were generated for this^ purpose. * 

* * 

, A new set of parameter estimates was determined from the simulated 
test data using the LOGIST program, and these estimates were compared to 
those used to generate the data and also to traditional item statistics 
computed on the data. These correlations . art shown in Table 3. The pattern 
of correlations bbta'iaed f rom the analysis of the simulation data was very 
similar to that obtained* from the real data. • The only change in the pattern 
of significant 1 correlations is that for the simulation data, the a-parameter 
and c-parameter estimates were correlated while t^hey were not for the real data. 
Since the magnitude of the two correlations is fairly similar (.21 vs. .33) 
this may be a chance result. Note that the correlation between the a-parameter 
estimate and the itenvbi serial correlation is" low for these data, just as 
it was for the real data. This result may indicate that the unexpectedly 
small magnitude of this correlation is an artifact of the particular range 
of difficulty and discrimination present in this data-set. 



Table .3 

Correlations Between the True Item Parameters, 
,Es"timated Item Parameters, and Tradition Item Statistics 
for the Simulated Test Data^ 



Item Scatisti 



cs 





















a b 


c 


est. a 


est. 


b est c 


. ' P 


r BIS - 


1 

a 


..'54* 


.20 


.69* 


.5*4* 


-.00 


-.47* 


.05 


b 




.22 


.42* 


..98* 


.08 


-'.94* 


-.44* 


» 

c , 






.23* 


.14 


> .75* 


-.01 


-.30*' 


est,, 


a* , 




> 


.47* 


.33* 


-.29 


.10 J 


est. 


b 








.08 


-.92* 


-.39* 


est. 


c 










.14 , 


-.33* 


P 






. 1 








.47* 


rBIS 












1 * » * 





Note: The correlations with astericks are significant beyond^ the .05 level. 
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The frequency distribution for the residual covariance -matrix for the 
simulated data is given in Figure 2 along with the descriptive statistics. 
The distribution is fairly symmetric around -;01 and does not have the long 
positive tail present in* th$ distribution for the real data. The standard' 
deviation of the residual covariances is .03, exactly what would be predicted 
based on a hypothesis of nonlinear relationship among the residuals. "This 
should be contrasted with the* y big differences between the expected and 
observed standard deviations for the real 'data, 

Two-Parameter Logistic Analysis The 2PL analysis of the real data' 
was performed using the MAXLOG program (McKinley and Reckase, in press), 
which was written specifically for test analysis using the M2PL model. 
Since the 2PL model is the .unidimensional' case' of the M2PL model, the 
program applies equally well for ,a one^dimensional solution. The item 
parameter estimates for the 2PL iSqdel are given ,in Table 4.* In addition to* 
tiie a- and d-parameters of the model) this table presents the b-parameter 
estimates for those individual^ that arc. 1 more familiar with that form of 
the 2PL model. ' * . ' # 

Both, the d- and b-parameter estimates from the 2PL analysis roughly 
maintain the order of ihe; items as shown by the proportion correct difficulty 
values shown in Table 1. Th e a-values also seem to % be' r elate d to t he 
difficulty of the items iri .that the higher discrimination parameter estimates 
were 'obtained for the easier items, The correlations of the 2PL parameter 
estimates and the'3PL and traditional statistics, shown in Table 5, support 
this observation. The 2PL a-parametjer estimates are correlated .66 with 
the d-parameter estimates, -.54 witft the b-parameter^ estimates , and .59* 
with the p-values. \ 

' * l \ 

The 2PL a-j;arameter estimated are clearly Wore closely related to the 
traditional concept of item discrimination than dre the 3PL a^parameter . 
estimates. The 2PL a-parameters correlate .95 with rg Ig and only .09 witl\ 

'the 3PL a-parameters. The difference in the two IRT discrimination estimates 
seems to be mainly in the values computed for the hard items and those with 
high guessing. Items 16, 30, 42, 44,^46, 48, 49 and 50 are good examples \ 
of the differences present in the two types of estimates. 4 \ 

The 2PL a-parameter estimates also correlated with the 3PL c-parametejr \ 
estimates. The. -.34 correlation obtained is consistent with the idea fchat \ 
items^ with high guessing levels should be low discriminators. The d-parameti 
estimates correlate highly with the 3P,L b-parameters and the .p-values . ! \ 
There are also smaller correlations with the measures of discrimination, j j 

i 

The frequency distribution and the descriptive .statistics for the 2PL 
residual covariance matrix are shotfn in Figure 3. The distribution looks 
very similar'to the* 3PL residual covariance matrix and the descriptive 
statistics are virtually identical. Based on these results, the two models 
would be considered to fit the data equally well. 1 
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Figure 2 . 

Frequency Distribution of the Reqiciual Covariances 
for the 3PL Model Applied to_the Simulated Data 




COVfiRIfiNCE 



U 



x 7 = -.01 
SD ■ .03, 
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Table 4 



MAXLOG 2PL Item Parameter Estimate 
for the AERA Symposi&m Data 



* Parameter 



Item 

« 


■ 

a 


d 




b ^ 


i 


1/35 


3.20 




-2.37 


2 


.82 


2.05 




-2.50 


3 


.97 . 


2.10 




-2.16 


4 


1.93 


2.72 




-1.41 


5 


1.66 


2.29 




-1.38 


6 


.59 


■1.59 




-2.69 k 


7 


* 1.66 < 


" 1.97 




-1.19 


8- 


2.02 


2.18 




-1.08 


9 


1.90 


2.09 




-1.10 


10 


0.88 


1.51 




-1.72 


.11 


1.41 


1.65 




-1.17 


12 


1.58 


1,67 




-1.06 


13 . 


1.29 


1.48 




-1.15 


14 


1.01 


1.35 




-,1.34- 


15 1 ' 


1.22 


1.40 


- 


-1.15 


16 


• 94 


> 

1.24 




-1.32 


17 


2.30 


1.85 ' 




- .80 


18 " 


.89 


1.12 




-1.26 


19 


1.12 


1.19 




-1.06 


20 


.63 


.99 




-1.57 


21 


1 . 48 


1.22 . 




- .82* 


22 


1.75 


1.32 




•- .75 


23 


1.07 


1.04 


- 


- .97 - 


24 


.87 


.87 




-1.00 


25 


1.69 


1.11 




- .66 


26 


1.44 


.86 




. - .60 


27 


1.06 


.75 , 




- .71 


28 


1.09 


' ' ».69" 




- .63 


29 


1.61 v 


.80 




- .50 


30 


.58 ' 


.■31 




- .53' 


31 


1.51 


.40 




- .26 


32 


.35 


.25. 




- .71' 


33 


1.10 


.22 




- <20 


34 


~r :n ■ .97. 


.15 




- .15 


35 


1 1.21 






- .08 


'36 ' 


U33 






.13 


37 


.'46 . 


19/' 




.41 


38 


• .92 


-. 2"6 




.28 ■ 


39 


.96 


-. 27 




.28 


40 


1.06 


: ,33 




.31 



~\ 

• -Id 
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Table 4 (Continued) 

MAXLOG 2PL Item Parameter Estimate 
for the AEI& Symposium Data 



Parameter 











Item, 


a 


d 


b 


\l 
















41 


.60 


- .57 


•95 . - / . 


42 ' 


.81 


- .60 S 


.74 


43 


* .81 


- .74 


.91 


44 


.65 ' 


- .79 


1.22 


45 


.80 


-1.00 


1.25 




.43 


-1.04 


2.42 


47 


.81 


-1.04 


1.51 


48 


.58 


-1.24 


2.14 


49 


.46 


-1.30 


2.83 


50 




-1.79 


9.42 . 




J \ « 






X 


1.10 " 


^ .69 


- .26 


SD 


.47 


. r. 17 ---^ 


1.83 



Table 5 

Correlations Between the Tradition Item Statistics 
and the 2PL and 3PL Parameter Estimates 
for the AERA Symposium Data , 



Item Statistics 





2PL-a 


2PLrd 3PL-a 


3PL-b 


3PL-C 


. P 


r BIS 


2PL-a 




.66* .09 


- .54* 


- 34* 


.59* 


.95* 


2PL-d . 




- .38* 


- .93* 


- 04* . 


. 98* 


.59* 


3PL-a 






.54* 




- .47*- 


.0* 


3PL-b 








.24 


- .34* 


- .50* 


3PL-c 










.03 


- .30* 


P ' 










• 


.54* 


r BIS 




/-A 











^ Note: The correlations marked by asterisks are significant 

ERIC beyond the .05 level. J[y 
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Figure 3 



Frequency Distribution of the Residual Covariances 
for the 2PL Model Applied to the Reai Data 




COVPRIfiNCE 



x = -.01 
SD - .04 
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Two-Dimensional Analysis 

/ * 
Since the standard deviation of the residual covariance tnatrix was 
larger than wouLd be expected if the models fit the data fcr both the 3PL . 
and 2PL models, it was assumed that a higher dimensional solution was 
required. The M2PL model was, therefore, rune on the AERA symposium data 
assuming a two-dimensional solution. The item parameter estimates from 
this analysis are given in Table 6. Notiq^ the d-parameter estimates still 
decrease uniformly with the increase in the item number, showing that the 
item difficulty estimates are still closely related to the p-values. A 
cursory study of, the a-parameter estimates will show that in many cases 
they are very Similar. The estimation program does not place any constraints 
on the relationship of the two dimensions being estimated and as a result, * 
all of the a-parameters could be the same if the data being analyzed so 
indicated. 







Table 6 






Item Parameters from a Two 


-Dimensional 




MAXLOG Analysis 


of the, AERA Symposium Data 








< 


- 




Parameter 




item 


a i 




7 

• 


1 


1 • U D 


1.39 


2.72 


2 


..95 


1.56 


t. 01 


o 

•J 


.89 


1.30 


1.90 


4 


1..94 


2.37 


2.04 


5 


1*. 65 

' .85 


2.00 


' 1.77 ' 


6 


.80 


1.54 


7 


1.91 


2.38 


• 1.55 


8 


'^2.36 


2.50 


1.56 


9 


2.31 


2.50 


1.55 


10 


1.00 ' 


1.83 


1.49 


11 


1.63 


2.50 


1.53 


12 


1.71 


2.22 


1.30 


13 


,1-55 


2.04 


' 1.24 


14 


1.21 


1.70 


1.20 


15 


1.54 


2.01 


1.19 


16 , 


1.34 


1.44 


- 1.10 


17 


2.50 


2.50 


1.20 


18 


1.47 


1.44 


' l v *KQ2 


19 


1.43 


2.08 


1.05 


2p 


r , .78 
V ' 1..80 


1.04 


.92 


21 


2.47 


.97 . 


22 


2.50 


2.50^* 


1.00 


23 


1.41 ' 


1.70 


\ .88 


24 


1.08 


1.41 


\.64 


25 


2.50 


2.50 


\82 



lb • 
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Table 6 (Continued) 



Item Parameters from' a Two-Dimensional 
MAXLOG Analysis of the AERA Symposium Data 



\ 



Ite 



Parameter 



26 


1.8ff, 


2.41 


'.64 


fJ27 


> 1.20 


2.26 


.66 


28 


1.40 


2.04 


.56, 


29 , 


2.50 


2.50 h 


.60 


30 • 


. 1. 14 


-.97 ' 


.29 


31 


2.50 


2.50 


.-25 


32 


.48 


.48 


.23 


33 


1.98 


2.04 


.13 


34 


1.64 


1.65 


.08 


35 


2.50 


2.50 ' 


.01 


36. 


2.50 


2.50 


- .24 


37 


.62 


.64 


- .21 


38 


1.86 ~" 


1.70 


- .30 


39 


1.37 


1.5-9 


- .33 


40 


1.58 


2.34 


- .46 


41 . 


.94 


1.04 


- .60 


42 ' 


1.86 


1.37* • 


- .66 


43 


1.36 


1.24 


- .76 


44 


1.77 


.99 


- .94 


45 


1.37 


1.25 


-1.02 


46 


1.30 


1.42 


-1.27 


47 


1.31 


1.24 


-1.23 


48 


1.01 


.75 


-1.28 


49 


1.08 


.40- 


-1.48 


50 


,38 


.03 


-1.86 



X 

SD 



1.54 
.57' 



1.70 
.70 



.50 
1.06 



In order to gain a better understanding of the parameter estimates 
obtained from the two-dimensional solution, the estimates were correlated 
with those obtained from the 3PL and 2PL analyses. These correlations are 
shown in Table 7. The correlations between the ability estimates obtained 
from the M2PL, 3PL, and 2PL models were also computed. These correlations 
are given in Table 8. 



.16 
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m Table 7 

Correlations between the M2PL/3PL, and 2PL 
Item 'Parameter Estimates , 



Parameter Estimates 





' M2PL 






3 PL 




• 2PL 




• 

Model Parameter *a x 


•a 2 


d 


a 


b 


c 


* a 


t 

d 


4 

M2PL , a 2 - 


.81* 


.16 

* 

.52* 


:40* 
-.04 


-Ajf 
-/50* 


-.24* 
- . 33* 


.80* 

.< . 88* 


.22 
. 55* 


d ' ' 






-■<»* 


J; 94* 


-.00 


.60* 


.99* 


a 


\ 






' .54* 


.21 


.09 - 


.38* 


3PL , b 










.24 


-v$4*^- 


.93* 


c 




4 ' 








-.34* - 


.04 


2PL a 

a 




t 

1 










!'. - 

166* - 



/ 

Note: The correlations with asterisks are significant 
beyond the .05 level. ' 



1/ 



/ 
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Table 8 



Correlations between the M2PL, 3PL, and 2PL 
Ability Parameter Estimates . . 



• 


• * 

Parameter 

i 


• t 

Estimates 

— « 






* • 


M2PL ' ; 


3PL 


2PL* 


< 

1 


Moffel Parameter 


✓ * 

01 8 2 


8 ' ■ X 


• 6 

V 


r 


M2PL . 


-.91 


.14 


.20 






i ■ • 


.18 , > 


.13 




3PL * * 6 




o 


* i 
.97 




2PL . 8 










Note: Because of the large sample 
are significant. „ * 


size, all correlations 
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The correlation between the d-parameter estimates from the M2PL - 
two-dimensional solution and the d-parameter estimates from the 2PL model 
(.99) indicated that tljis parameters essentially the same for the two 
models. The a-parameters have changed, however. While the a x - and 
a 2 ~parameter estimates were highly related to each other and to the 2PL 
a-parameter estimates, they had quite a different relati>ortship with the 
item difficulty parameters and with the 3PL a-parameter' estimates . The 
ax-parameter wa^'sttmewhat related to the 3PL a-parameter, while t&e / * 
a 2 -parameter was not,. The a 2 -parameter was related to the difficulty 
parameter estimates, while was not. It would seem then that the M2PL 
a,-parameters were dividing up the variation in the parameters of the 3PL 
model. Both &i and a 2 were related to the 3PL c-parameter. 

»• 

The high correlation between the a^.and a 2 -parameter estimates is of 
special note. When these parameters are the same, it indicates that the 
tyo ability dimensions are required in eqyal proportions in responding to 
the test item. If all of the ax-parameters were equal to the a 2 -parameters , 
only one dimension would l>e needed in the model — that dimension would be 
thte sum of Q 1 and 8 2 . When a 1 and a 2 ar;e different x for an item, different 
amounts of ability on each dimension are required. 

The correlations between the ability estimates gave very interesting 
results, First, the M2PL ability estimates had a very high negative correlation 
(-.91)* This fact, combined with the similarity of^the a-pai?ameter estimates^ 
indicates that the exponent of the model can be approximated by d +a 1 8 1 -a 2 6 2 . 

1.8 ' ■ • 5 
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Since a t and a 2 are equal in many cases and so are and 8 2 , the value of 
the exponent is mainly controlled by the d-paranieter (the a 1 0 1 -a 2 0 2 term is 
zero). This would seerp to indicate that'the data have one predominant 
dimension shown by the change in difficulty of the items, and that the 
effect of the other dimensions is minor. % * 



A second interesting. result is that the M2PL ability estimates have % '? 
relatively low relationships with the 3PL and 2PL ability estimates. Their 
are clearly an indicetor of something different. . The Ability estimates 
from the 3PL and 2PL models are highly related to each other, as would be 
expected. . 

i 

The, fact that the M2PL item' parameters and person parameters w$re 
highly interrelated suggests that the data may be predominantly unidimensional. 
iy further check this hypothesis, the frequency, distribution of thl residual 
povariance matrix was formed for the two-dimensional solution for the M2PL 
mode>l. This distribution is shown in Figure 4. This distribution was alfeo • 
produced for the application of the M2PL model to the one-dimensional ^ 
simulation data in#order to have a basis for comparison. This distribution " 
is shown in Figure 

A comparison of these two distributions with those for the unidimensional 
models does not show any reduction iit the variation of the residual covariances 
In fact,^the distribution fo$r the unidimensional data has a larger "variance * 
when the M2PL model was used than when the 2PL model was used. This fact 
may syggest that the estimation of more parameters induces greater amounts 
of error in the parameter estimates. 

Although there was no reduction of variances iBv the M2PL distribution 
when compared to* the 3PL and 2PL distributions, there* was a shift in the 
mean. Both of the unidimensional models yielded a means residual covariance 
of -.01. The M2PL model had a jnean residual covariance of .02. While this 
difference is only .03, it is highly statistically significant (z = 26.25) 
because of the large number of observations. This larger mean may mean a 
pooler fit.tp the data for the M2PL two-dimeiysional model than for the 3PL 
or 2PL models^ Because of lack of experience in interpreting the residual 
covariance matrix;* this cannot be said for "sure.* 

The Identification of Item Clusters * ' 

0ne N of the purposes for performing the analyses on the data supplied * 
for this symposium was to determine whether the M2PL procedure could be 
used to sort itfems 'into homogeneous clusters. Based on past experience 
with the 3PL model (Reckase, 1977), it was felt that items that had high 
a-values on both dimensions of the M2PL solution would be good .measures of 
the dimension defined by, the d-parameter. Those items that had' low a-values 
probably measured some other dimension, and other combinations of a-values •» 
may indicate other item clusters. 

To operationalize the above ideas, a cluster analysis was run using 
the two a-parameter estimates as observation^. The Euclidean distance was v" 
ufced as a similarity measure. The BMDP1M frtfixon and Brown, 1977) program, 
was used for the analysis. , s The cluster analysis resulted in four fairly 
distinct sets of items, although more clusters could certainly be obtained 

if they were; thought warranted. The foyr clusters are shown in Figure 6. 

> 




x * .02 
SD - .04* 



Zv 
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The first cluster is made up of items that had high a-values on both 
dimensions. This cluster is very tight and distinctively different than 
the other itdm clusters present. The second cluster is made up of items 
that had fairly high discrimination parameter estimates for the second 
dimension, and slightly lower values for tUe first dimension. Recall that 
the second dimension was the one that had the a-values related to the item 
difficulty level.' ' 

s ' " * * 

The third cluster is composed of items that had middle range a-parameter 

estimates on both dimensions. The fourth cluster is composed of items that 

had relatively low discriminations on both dimensions. Item 50 was included 

with this cluster even though it seems like an outlier. It was the most 

difficult item on the test. Although these clusters seem reasonable from a 

statistical point o$ view, without knowing the content of the items it is 

impossible to tell the variables that control cluster membership. 

Discussion 

When the analysis of the AERA symposium test data was begun, the 
anticipated result was that two or'more relatively* distinct ability dimensions 
would be discovered and that the items on the test could be classified -into 
content categories based on which dimensions were required for successful 
performance? This was not found to be the cas A e. Rather than distinct 
abilicy dimensions, two highly correlated dimensions we r^ developed. 
Rather then finding that the M2PL model fit the data better -than the . 
unidimensional models, the fit was found to be about equal^ if not worse. 

Two possible conclusions come to mind based on these results. .The 
first is »that the estimation program for the M2PL model, or the model 
itself is inadequate. Although this is certainly a possibility, the fact 
that the estimation procedures did a goo3 job of recovering true parameters 
in simulation studies and estimating parameters for fnultidimensional test 
data (McKinley, 1983) would argue against such an interpretation. 

The second conclusion that comes to mind is th^it the test data really 
do hav^ predominantly one dimension and that the results of the analysis 
reflect tha*t fact. Even if the test does require multiple abilities, this 
fact may be clouded by 'the range of item difficulty present in the items. 
This may be a case similar to a test made up of three items: jump over a 
string, define "precipitation, 11 and solve a differential equation. Although 
these items) measure distinctly different skills, for *a population that 
ranges widely in ability, the items will appear statistically to measure a 
single dimension. We anxiously await the information about the nature of 
the -data the we have been analyzing to determine which conclugion is correct. 
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